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Introduction 

Until relatively recently, data on the workplace 
and main workforce of our education system, 
schools, teachers, and administrators, have cot 
been available on a regular basis. The Schools 
and Staffing Survey(SASS) was designed to meet 
the need for information on the characteristics and 
experience of teachers and administrators, to 
describe the essential features of the school as a 
place to work and a place to learn, and to provide 
data on aspects of teacher supply and demand, and 
attrition. The SASS was first fielded in the 1987- 
88 school year, was repeated in the 1990-91 
school year, and is intended to be conducted every 
three years. 

The SASS is a complex undertaking, both in 
design and in implementation. Because of its 
complexity and the fact that it is a regular 
recurring program of the National Center for 
Education Statistics(NCES), the SASS staff realize 
that opportunities exist to learn from the successes 
and failures of each collection cycle. These 
lessons provide the opportunity to make 
improvements to the program as well as gather 
information on the quality of the survey’s data 
products. 

This paper is intended to serve two purposes: 1) 
to provide a brief overview of the SASS program; 
and 2) to identify areas of research or survey 
evaluation in which work is ongoing or planned. 



Overview of the SASS 

The SASS is an integrated system of surveys of 
public and private schools, school districts, school 
principals and administrators, and teachers. The 
data collection program consists of seven mail- 
out/mail-back surveys implemented during one 
school year, followed by a mail survey of a 
subsample of teachers one year later. These 
surveys include: 

1 . a survey of public school districts(local 
education agencies); 

2. a survey of schools, public and private; 

3. a survey of school administrators/principals in 
the public and private sectors; 

4. a survey of teachers in the public and private 
sectors. 

In ihe school year lolloping the implementation of 

the SASS, a subsample of teachers in the SASS 



teacher survey are selected to be included in the 
SASS Teacher Followup Survey. This mail 
survey, a survey of public and private school 
teachers, was designed to provide information on 
teacher attrition and retention in public and private 
schools. 

The SASS program has also included a research 
sample as part of its ongoing operations, thus 
providing opportunities to study questionnaire 
design, content, operational and survey methods 
issues in the context of a large scale operation. 

The SASS is a broad multi-dimensional program, 
guided by four principal objectives: 

1. to provide data on the components of teacher 
supply and demand, shortages and turnover, and 
the policies and practices influencing supply and 
demand. 

2. to provide data on the principal/administrator 
workforce, including demographics and economic 
characteristics as well as their academic 
background, qualifications, and experience. 

3. to provide data on teachers, including 
demographic characteristics, academic background, 
qualifications and experience, teaching 
assignments, workloads, and compensation. 

4. to provide data on school conditions and 
programs, including enrollments, staffing, 
organization, teaching load, problems and locus of 
control. 

The SASS accomplishes these objectives through 
a design that allows the development of state and 
national estimates for public schools and affiliation 
and national estimates for private schools. 
Schools are the primary sampling unit, and a 
sample of teachers, on average between four and 
eight, is selected in each sample school selected. 
Public school districts are included in sample 
when one or more schools in the district are 
selected. The following are sample sizes for the 
1990-91 SASS: 5400 local education agencies; 
13,200 schools (9,900 public and 3,300 private); 
13,200 school principals/administrators (9,900 
public and 3,300 private); and 65,200 teachers 
(56,000 public and 9,200 private). 

By the nature of its content and design, the SASS 
provides opportunities to address issues on 
education policy. The existing SASS data linkages 
among the levels of the education hierarchy - - 
teachers, principals, schook and school district- - 
and the potential to add several mote, namely 
students and parents, indicates the importance of 
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the SASS in the elementary and secondary 
education statistics program in the National Center 
for Education Statistics. With this in mind, this 
paper presents the current operational and research 
issues of the program. The topics covered are: 
l)frame and coverage issues; 2) questionnaire 
design; 3) data collection; 4) estimation and 
nonresponse; S) measurement issues; and 6) 
evaluation of survey data. 

Frame and Coverage Issues 

The sampling frames for the school component of 
the 1990-91 SASS are the "public school universe" 
of the NCES’ Common Core of Data(CCD) and 
the Private School Universe Survey(PSS). The 
CCD contains statistical information collected 
annually on all public elementary and secondary 
schools and school districts in the nation. State 
Education Agencies(SEA) compile and transmit 
data they collect from schools and school districts 
into formats defined by NCES. Information they 
provide includes school or district name, address, 
school type, enrollment and student characteristics, 
and the number of classroom teachers. 

The Private School Universe Survey(PSS) is a data 
collection aimed at building an accurate and 
complete list of private schools in the U. S. The 
schools on the PSS come from a combination of 
private school lists and area frame searches. The 
PSS contains data on religious orientation, level 
and size of school, enrollment, number of 
graduates, and number of teachers employed. 

CCD - SASS Differences 
The 1990-91 SASS represented one of the first 
uses of the CCD for sampling purposes. School 
data in SASS were collected using the definitions 
established in CCD. However, an understanding 
of the relationship of these definitions to 
principals’ and administrators’ understandings of 
our concepts was limited. Furthermore, because of 
the time needed to edit several CCD variables and 
the time needed to draw and prepare the sample, 
the 1988-89 CCD was used to draw the 1990-91 
SASS sample. Consequently, a number of schools 
in the SASS sample reported teacher counts and 
student enrollment counts that differed from the 
CCD file. Timing, school reorganization, CCD 
misreports and definition misunderstandings may 
play a role. A project is underway to understand 
these differences by characterizing the schools 
with discrepant information. 

Evaluation of the Private School list Building 
The sampling frame for private schools is a 
combination of list and area frame samples, the 
latter being necessary to compensate for the 
known undercoverage ot the list frame. The 
private school list frame is composed of private 



schools contained on a commercially available list 
from Quality Education Data(QED) , private school 
associations, state departments of education, and 
other sources listing private schools. Recognizing 
the large undercoverage of the list frame in the 
previous PSS(approximately 20%), the 1991-92 
PSS made a substantial effort to acquire additional 
private school lists from the states. A project to 
evaluate this effort and the impact of these new 
sources is underway. 

Evaluation of the Teacher Listing 
The sampling of the teacher component of the 
SASS requires schools in the SASS sample to 
provide a list of teachers in their school along 
with the following information: whether new or 
experienced; race/ethnicity; bilingual/English as a 
Second Language(ESL); field of teaching. The 
issue for the SASS program is whether the school 
is filling the teacher list properly. SASS data have 
shown an inconsistency in the number of teachers 
listed by the schools during the listing operation 
early in the school year versus the numbers 
provided later in the year on the school 
questionnaire. A study is being developed that we 
expect will improve our insight as to how teacher 
estimates can be improved. This study will 
involve a reinterview of the person providing the 
teacher lists. Reconciliation of differences in the 
original and reinterview list will provide an 
approximate idea of the "true" number of teachers. 
Also, under consideration is a reinterview and 
reconciliation of responses from the teacher listing 
operation and the school questionnaire. 

SASS Student Sample 

In recent years, interest has grown in augmenting 
file SASS program with a sample of students. 
Because SASS is a national probability sample of 
schools, SASS is capable of providing a national 
probability sample of students distributed across 
elementary and secondary grades. Other NCES 
school-based surveys do not yield such samples, 
because they are oriented to one or two grade 
levels. This sample could lead to the study of 
equity issues: which students are taught by 
better/worse prepared teachers? Which students 
are participating in various programs? The 
statistical issues with the student sample are the 
development of procedures to draw a sample of 
students at the school and the ability to correctly 
calculate a probability of selection. 

During the 1990-91 SASS, a research panel was 
fielded to address the issue of how a student 
sample should drawn - whether by an employee 
of the school or by the Census Bureau. 
Frazier(1992) reported on the results of this test 
and found that it was difficult for an untrained 
school employee to correctly drau the sample. 
This field test also showed that the questions 
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aimed at determining the probability of selection 
did not work well. Thus, NCES and Census 
Bureau staff continue to work on this problem 
with a view to implementing another pretest in 
early 1993. 

Expand Survey Coverage 
The SASS target population is limited in two 
ways. First, schools offering only kindergarten 
and prekindergarten classes are defined to be out- 
of-scope in the SASS. This is a serious limitati on 
given the current strong interest in policy issues 
related to early childhood education. 
Furthermore, NCES has no sampling frames that 
adequately cover the prekindergarten and 
kindergarten programs in the public and private 
sectors. During the next year, NCES plans to 
study alternative approaches to improving the 
coverage of prekindergarten and kindergarten 
programs. The adequacy of different sources of 
information on these programs will be reviewed 
and assessed. An exploration of the possibility of 
using the area search and area sampling 
approaches used in the PSS will also be reviewed. 
Second, the American Trust Terri tones (American 
Samoa, Guam, Virgin Islands, Northern Marianas, 
and Puerto Rico) are included in the CCD 
universe, but not in the SASS. Staff will explore 
the feasibility and cost of expanding SASS 
coverage to include the territories. 

Questionnaire Design 

SASS is a system of mail surveys with telephone 
follow-up. The surveys require teachers, 
principals, personnel officers, and administrative 
assistants to be able to respond to questions about 
complex concepts without much help. The 
complexity of the current forms and concepts offer 
many opportunities for the respondent to make 
mistakes. Several projects are now underway to 
address the shortcomings of the design and format 
of the cunent questionnaires. 

Cognitive Research Program 
Results from the reinterview program at the 
conclusion of the 1988 cycle of the SASS 
indicated a number of items required 
improvement(Bushery, Royce, and Kasprzyk, 
1992). The program's response was two fold: 1) 
to commit a substantia] amount of professional 
time to reviewing completed questionnaires) and 2) 
to conduct a cognitive research program in 
preparation for the 1990-91 SASS. Similarly, in 
preparation for the next cycle of SASS, a 
research program of detailed, probing interviews 
using the public school questionnaire was initiated 
in the fall of 1991. Jenkins(l992) reports on 
results of the latter study. This program of 
cognitive research will next focus on two 
questionnaires. First, the SASS student 



questionnaire will be reviewed and redesigned for 
testing in the spring of 1993. Second, the school 
questionnaire, already the focus of much review, 
will be redesigned, reformatted, and tested. 

Computer Assisted Interviewing 
The increasing availability of personal computers 
in schools and school districts suggests a potential 
application of computer assisted interviewing in 
the SASS. Since SASS is a mail survey, a 
suggestion has been made to consider sending 
diskette-based data collection instruments to 
schools and school districts as an alternative to the 
paper and pencil instruments. This data collection 
system’s potential for improving the data collected 
in SASS is significant. Automated range checks, 
edits, logical edits, and skip pattern checks 
provide opportunities to clarify reported data at 
their source — the respondent — immediately upon 
reporting. Plans are being developed to design, 
implement, and evaluate an automated data 
collection instrument. 

Teacher's Self-Report of Academic Background 
Hie 1987-88 SASS provided evidence of teachers 
having difficulty reporting their academic 
backgrounds. In response, two views of collecting 
these data have been proposed - by asking for the 
number of credits earned in critical subject areas 
or by asking for the number of courses taken in 
each critical subject area. During the 1990-91 
SASS, a small research panel, 200 schools, having 
a sample of 867 teachers was fielded to test these 
two approaches. As part of the test, we asked for 
a signed authorization from each teacher to allow 
the NCES to obtain transcripts from the schools 
he/she attended. Transcripts will be coded by 
subject area to allow comparisons to the self- 
reported data. 

Data Collection 

Several issues have arisen concerning the 
operation of the SASS data collection system. 
Ibese issues bear on the quality of the reported 
data and the improvement of the timing of the 
availability of the S ASS sample. 

Data Collection Mode 

SASS was designed to be primarily a mail- 
out/mail-back survey. Telephone followup was 
used for all sample units not returning the mail 
questionnaire. Because there is a substantial 
telephone followup( 33% for the public schools 
and 46% for the private schools), there is concern 
about possible response bias due to the mode. 
Parmer, Shen, and Tan(1992) address the issue of 
possible response bias by mode. 

Improving the Availability of the SaSS Sample 
Teacher sampling for the SASS requires the 



BEST COPY AVAILABLE 



development of a teacher list for each sampled 
school. These lists of teachers are requested from 
the schools in the SASS sample, checked in at the 
regional office for completeness, mailed to 
Jeffersonville where the total number of teachers 
of each type are keyed and transmitted to 
Washington. Washington then sends sampling 
instructions back to Jeffersonville where the 
specific teachers are selected. The data for the 
selected teachers are keyed and transmitted back 
to Washington. Washington then matches the 
teacher information to the school information to 
create the sample file. The sample file is then 
used to mail questionnaires to the teachers. 

This cumbersome process has an obvious and 
direct bearing on the timeliness of the availability 
of the teacher sample. A working group has been 
established to study the potential for improving the 
efficiency of this operation. 

Data Comparability Project 
Response burden is a concern for all federal data 
collections. The hierarchical design of the SASS 
and a number of individual items, particularly as 
they relate to school district staffing, have proven 
burdensome to SASS respondents, in response to 
the reactions of several school districts and in 
pursuit of better data collection methods, NCES 
developed a project to test whether state education 
agencies have the capacity to provide data from 
their automated record systems that would 
otherwise be collected in SASS from local 
education agencies. How comparable are the data 
available from the state education agencies to the 
data collected in the SASS from the individual 
local education agencies? Blank(l992) reports on 
the results of this project and its direction in the 
future. Successful collection of district level 
staffing items from a state automated record 
system would lead to major rethinking of SASS 
data collection methods for the teacher demand 
and shortage survey. 

Estimation and Nonresponse 

While unit response rates in the SASS are quite 
good, nonresponse remains a concern because of 
the hierarchical nature of the SASS design. 
Principals may often act as gate-keepers for the 
teachers in sample by not providing lists of the 
teachers in their schools; principals may complete 
the principal/administrator form but not the school 
form. Districts may also serve as gate-keepers for 
their schools. 

Nonresponse 

A project to investigate the characteristics of 
nonrespondents in the 1990-91 SASS is under 
development. Characteristics of SASS units - 
districts, schools, principals/administrators, and 



teachers - respondents and nonrespondents will be 
compared across many dimensions with a view to 
providing an understanding of nonresponse in the 
SASS. 

Work has also begun on assessing the nonresponse 
adjustment cells chosen for the SASS school 
survey and the associated cell -collapsing strategy. 
These cells had been selected based on intuitive 
analytic judgement. This study, however, is 
intended to quantify these judgments and propose 
alternatives if necessary. Some preliminary results 
for the school component of the SASS are 
available in Shen, Parmer, and Tan (1992). A 

replication of the study on other SASS analytic 
units - principals/administrators, school districts, 
and teachers - is also desirable. 

Plans are underway to increase the number of 
followups in the Teacher Followup Survey 
component of the SASS, thereby making this 
component a multiwave longitudinal study of 
teachers. Even though item nonresponse is 
relatively small in this survey, research on 
imputation methods that use previously collected 
data is desirable. 

Variance Estimation 

In SASS, the sampling unit is the school. School 
districts are brought into the sample because a 
school in the district has been selected in the 
SASS sample. Thus, the school district collection 
unit is an aggregation of schools(the sampling unit) 
belonging to the district. Kaufman( 1 992) 
addresses foe issue of how well balanced half- 
sample replication methods estimate variances 
when foe collection unit (school districts) is an 
aggregation of sampling units (schools). 

Generalized variance models provide data users an 
easy way of obtaining variance estimates for 
complex sample surveys. A project is under way 
to develop generalized variance models for each 
component of foe SASS. 

Measurement Issues 

Much attention has been given to resources and 
students as foe principal measures of institutional 
improvement — expenditures per pupil, for 
example. Statistics such as these, however, 
provide little information about school quality or 
foe quality of foe educational experience in 
schools. To remedy this situation, a number of 
ideas, as discussed in foe report of a Panel on 
Education Indicators (1991), will be developed as 
potential measures of educational experience and 
institutional quality. A series of research projects, 
field studies, and feasibility studies will be 
developed duiiug the next several y ears. The 

research will be a combination of conceptual 



research (appropriate measures), item and content 
research (the possible ways the measures can be 
implemented), and field and validation research 
(do these measures work in practice and do they 
work in large scale surveys). Originally 
conceived as a component of the SASS, the 
direction of the research may either lead to a new 
survey or significant modifications to an existing 
survey. 



p.vfllnatinn of Survey Data 

Evaluation of the quality of survey data can take 
several approaches. Microdata studies which 
evaluate the quality of the individual response, 
field performance statistics, experiments to test 
competing methodologies, and macrocomparisons 
with other established data sources are all used to 
establish the validity of a data set. SASS has 
several projects, ongoing and in the planning 
stages, which will bring information to bear on the 
quality of the SASS data. 

SASS Reinterview Program 
SASS has fielded a reinterview program in each 
cycle. Each reinterview was aimed at measuring 
simple response variance, a measure of the 
inconsistency between responses over repeated 
applications of the question. Thus, the purpose of 
the reinterview was to identify questions needing 
improvement in the next cycle of SASS. Bushery, 
Royce, and Kasprzyk(1992) describe results from 
the reinterview programs and show how these 
programs can be a tool for identifying problem 
items in a questionnaire. While the Bushery et al 
paper shows results from a limited number of 
questions in both the 1987*88 and 1990*91 SASS, 
an analysis of the quality of substantially more 
items is available in an internal 

memorandum(Royce, 1992). 

The 1991-92 Teacher Followup Survey (XfS) 
component of the SASS also conducted a 
reinterview to ascertain the quality of individual 
items. This reinterview program, however, 

featured the use of a probing, reconciled 
interviews to improve the reinterview’s diagnostic 
power. In this case, we expect to obtain 

information not only on questions that are 
unreliable, but also on the reasons for the 
inconsistency in responses. 

Comparing Estimates across Forms 
The SASS obtains the same or similar data across 
several survey forms. Thus, for example, it is 
possible to obtain rates of attrition and separation 
not only in the SASS but also in the TFS. 
National counts of teachers are available from the 
teacher, school, and district forms. Estimates of 
the number of certified teachers are available from 



both the teac her and district questionnaires. The 
relationship between these seemingly equivalent 
estimates is not well understood. During the next 
year, a project will begin to identify all estimates 
of the same phenomena across the different 
components, to quantify the differences if they 
exist, and to try to understand the reasons for the 
differences. 

Evaluating Self-Reports of Urbanidty 
In both the 1987*88 and 1990*91 cycles of SASS, 
the question, "Which best describes the community 
in which this school is located?” was asked of the 
principal(for the administrator/principal survey) 
and the respondent to the school survey. The 
response categories were given as rural, small city 
or town, medium-sized city, suburb of a medium 
city. etc. These reports are highly subjective and 
have exhibited moderate response variance as 
determined through the reinterview 
program(Bushery et al, 1992). Recently 
Johnson(1989) developed a methodology for 
assigning "type of locale" codes based on the 
school mailing address being matched to Bureau of 
the Census data files containing population and 
population density data, Standard Metropolitan 
Statistical Area(SMSA) codes, and a Census code 
defining urban and rural areas. A study is in 
progress to compare codes derived through the 
Johnson algorithm to the self-reported 
classifications found in the two cycles of SASS. 
These comparisons will give us a better 
understanding of this survey item. Since the self- 
report method is used on many NCES surveys, the 
results of this study have wader applicability than 
the SASS. 

SASS Quality Profile 

Work on developing a quality profile for the SASS 
has begun. The quality profile will summarize 
methodological and evaluation research related to 
the SASS and will provide an overview of 
procedures for all phases of the survey - sample 
selection, data collection, data processing and 
estimation. It is intended to provide an overview 
of what is known about the sources and magnitude 
of errors in the SASS, and thus a sourcebook of 
information on the quality of the SASS data. 

SASS User Survey 

In designing the SASS approximately six years 
ago, NCES anticipated a variety of users • 
education planners, policymakers, managers, 
government analysts, and academic researchers. 
By the end of 1992, SASS will have released 
several major reports, a number of E.D. Tabs, 
public use microdata tapes and CD ROMs, and 
restricted access data tapes. In February 1992, 
the SASS Review Board, a working group of 
researchers interested in the use and evaluation of 
the SASS, suggested the need for a SASS user 
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survey. The purpose of the survey would be to 
identify uses/users of the SASS and to assess 
whether they are consistent with the uses and users 
as identified in the design phase. The survey 
would also attempt to assess whether the available 
survey products meet user needs and how 
dissemination efforts could be improved. In the 
near future after the goals of survey are more 
clearly defined, we anticipate developing such a 
survey. 

Data Analysis: Quantity and Quality of T e a cher 
Labor Supply 

This year an analytic project that focusses on 
estimating the effects of compensation and other 
policy variables on the quantity and quality of the 
teacher labor supply will begin. This project will 
address: 1) the estimation of the external labor 
supply facing schools and the effect of 
compensation and other school-level variables on 
this supply; 2) the estimation of internal labor 
supply, i.e. the retention of the teaching workforce 
and the effect of compensation and other policy 
variables on temporary and permanent flows in 
and out of the teaching profession. While a 
decidedly substantive analytic project, this project 
is intended to also re-evaluate the vacancy matrix 
data on the 1987-88 Teacher Supply and Demand 
Questionnaire, data that were not released on the 
public use tapes due to response inconsistencies. 



Fndnnte 

Any large complex data collection raises numerous 
questions about methods and data quality. SASS 
is still in its infancy in terms of understanding and 
use. The 1990-91 SASS included a methods 
research panel to help assist in answering 
unresolved methods issues. The 1993-94 SASS 
will also include such a panel. This commitment 
as well as the research commitment described in 
the projects above will provide a much deeper 
understanding of the SASS data as well as 
improve the quality of the survey operations. 
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1. INTRODUCTION 

The Schools and Staffing Survey (SASS) is a 
good example of how a reinterview program can 
contribute to improved data quality by identifying 
questions which need improvement. We believe we 
have improved one aspect of SASS data quality, 
simple response variance - in part because the SASS 
reinterview program identified questions needing 
improvement. 

The 1991 SASS reinterview results also suggest 
that mail respondents provide more reliable data than 
those interviewed in a telephone follow-up operation. 

1.1 The SASS Surveys 

The National Center for Education Statistics 
(NCES) sponsors, and the U.S. Bureau of the Census 
conducts, the Schools and Staffing Survey (SASS) to 
provide data on teachers, school administrators, 
schools, and local education agencies. 

The SASS runs on a three-year cycle, the first in 
1987-88 and the second in 1990-91. The Census 
Bureau conducts the SASS by mail, with telephone 
follow-up of cases not responding by mail 

Mail response rates range from 49 percent (for 
private schools) to 80 percent (for public school 
administrators), with final response rates between 83 
(private school teachers) and 97 percent (public 
school administrators again). We completed one- 
sixth to one-third of the cases using telephone follow- 
up. 

1 .2 The SASS Reinterview Program 

Two major purposes of reinterview programs are 
quality assurance and estimating response error (1]. 

The SASS reinterviews estimate simple response 
variance, a measure of the inconsistency between 
responses over repeated applications of a question. 
Our main goal is to identify questions needing im- 
provement for the next cycle of SASS. We identify 
problematic questions in the reinterview and follow 
up with cognitive research and other questionnaire 
design techniques to make the improvements. 



To estimate response variance accurately, the 
survey error model assumptions require the reinter- 
view to be an independent replication of the original 
interview. Independence is difficult to achieve 
because the respondent might remember his or her 
answer to the original interview question. To the 
extent a reinterview lacks independence, response 
variance may be underestimated. Operational con- 
straints often make it difficult or impossible to 
conduct the reinterview as an exact replication of the 
original interview. When a reinterview does not 
replicate the original interview perfectly, the differ- 
ences in methodology may overstate the response 
variance. 

The SASS reinterviews fail to replicate the 
original interview in two respects: 

• All SASS reinterviews contained fewer questions 
than their original counterparts. 

• The original SASS surveys used self-administered 
mail-return questionnaires (with telephone follow-up 
of non-respondents). Except for the 1991 SASS 
School Survey, all the reinterviews were conducted 
by telephone. 

We conducted the Census Bureau’s first-ever mail 
reinterview in the 1991 SASS School Survey. Some 
of the 1988 SASS reinterview findings suggested that 
for some questions, the reinterview model assump- 
tions were not adequately met [2], Section 2.3 
discusses this topic in more detail. These results 
prompted us to evaluate the 1991 SASS School ques- 
tionnaire through a mail reinterview. 

1.3 Response Variance Measures 

Response error consists of response variance ind 
bias. The Census Bureau estimates two main metrics 
(from unweighted data) to quantify response variance, 
the gross difference rate and the index of inconsis- 
tency. In a categorical variable, one-half the gross 
difference rate equals the simple response variance. 
The gross difference rate also represents the propor- 
tion of respondents who change their answers from 
one interview to the next. In a question with a gross 
difference rate of 20 percent, one fifth of the respon- 
dents changed their answers. 

The index of inconsistency is a relative measure 
of response variance. A simplified definition of the 
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index is the ratio of the simple response variance to 
the total variance of a characteristic. The L-fold 
index of inconsistency is a weighted average of the 
indices over all categories in a multi-category ques- 
tion. An index of 50 means that half the total vari- 
ance of a characteristic can be attributed to response 
variance. Experience provides a rough rule of 
thumb for interpreting the index of inconsistency. If 
the index is: 

• less than 20, response variance is low. 

• between 20 and 50, response variance is moder- 
ate. 

• greater than 50, response variance is high. 

High response variance means the question itself 

causes at least as much of the variability in the data 
as the variability among respondents in the popula- 
tion. Two reasons for high response variance are: 

• The question is poorly worded and confuses the 
respondent. 

• The information requested is too difficult for the 
respondent to provide. 

Because the index of inconsistency estimates the 
ratio of two variances, the index itself has high 
variability. If the data don’t provide enough cases in 
each original-by-reinterview outcome cell, a reliable 
estimate of the index cannot be computed. 

2. REINTERVIEW RESULTS 

This paper compares response variance results for 
questions reinterviewed in both the 1988 and 1991 
cycles of SASS. Table 1 shows reinterview sample 
sizes and completion rates for 1988 and 1991. We 
used unweighted data and tested all comparisons at 
a = 0.10. Tables 3 through 6 display 90 percent 
confidence intervals in parentheses. 

The Administrator and Teacher surveys ask both 
attitudinal and factual questions. In 1988 the attitudi- 
nal questions we reinterviewed showed high levels of 
inconsistency [2]. Inconsistency in attitudinal ques- 
tions may result from rimple response variance or 
from actual changes in attitudes between the original 
interview and reinterview. In 1991, we decided to 
concentrate the reinterview on factual questions — 
with the aim of improving future cycles of the SASS. 

In the 1988 SASS, we could estimate the index of 
inconsistency reliably for 35 of the 45 factual ques- 
tions we reinterviewed. We estimated the index 
reliably for 109 of the 126 factual questions reinter- 
viewed in 1991 [3]. Table 2 summarizes the results 
of both SASS reinterviews. 

Keep in mind that the distributions in Table 2 are 
not strictly comparable. We purposively selected 



different sets of questions for the two reinterviews. 
We evaluated 15 factual questions common to both 
cycles of SASS. Eleven of these questions received 
significant revisions in 1991. Four of the revised 
questions displayed reduced response variance. Our 
question improvement efforts have paid off, at least 
partially. 



Table 1. SASS Reinterview Sample Sizes 



1988 1991 

Administrator Survey 

Eligible for Reinterview 1309 1123 

Response Rate 87% 87% 

Teacher Survey 

Eligible for Reinterview 1126 1101 

Response Rate 75% 74 % 

School Survey 

Eligible for Reinterview 1309 1123 

Overall Response Rate 87% 84% 

Attempted Mail Reinterview 50% 

Percent Completed by Mail 43 % 

Attempted Telephone Reinterview * 57% 

Percent Completed by Telephone 41% 



* Includes 80 reinterviews not returned by mail 
and 85 original mail interviews returned too 
late for mail re interview. 



Table 2. Surmary of SASS Reinterview Results * 
Response 



Variance 


1988 


1991 


All Three Comments 




Low 


4 (11%) 


43 (39%) 


Moderate 


14 (40%) 


38 (35%) 


Nigh 


17 (49%) 


28 (26%) 


Actarinistrator and Teacher Surveys 


Low 


4 (19%) 


26 (36%) 


Moderate 


8 (38%) 


26 (36%) 


Nigh 


9 (43%) 


21 (29%) 


School Survey 
Low 


0 ( 0 %) 


17 (47%) 


Moderate 


6 (43%) 


12 (33%) 


Nigh 


8 (57%) 


7 (19%) 



* Questions for which index could be reliably 
estimated. 



2.1 Administrator and Teacher Survey Results 

The two Administrator questions reinterviewed in 
both SASS cycles ask whether the respondent earned 
a bachelor’s degree and a master’s degree. These 
"degree earned" questions are virtually the same as 
the corresponding Teacher survey questions. The 
results for Administrators were nearly identical to the 



Teacher results shown in table 3. The 1988 question 
provided a list of possible degrees and asked the 
respondent to "mark all that apply." The 1991 
question asked, "Do you have a bachelor’s degree?" 
If "Yes," the next question asked "Do you have a 
master’s degree?" The remaining degrees (associate, 
doctor’s, etc.) used a "mark all that apply" approach. 
Table 3 suggests the direct question format produces 
more reliable data for degree earned. 



Table 3. Teacher Survey Reinterview Results 
•* Degrees Earned 





1988 


1991 




Bachelor's Degree 






Percent Yes 


97.6 


98.1 




COR * 


7.5 


0.6 






( 6.0 - 9.2) 


( 0.3 - 


1.3) 


Index 


79.5 


Too few 


cases 




(64.2 - 98.5) 


did not mention 


Master's Decree 
Percent Yes r 








41.5 


41.4 




GDR * 


4.3 


1.1 






( 3.2 - 5.7) 


( 0.6 - 


1.9) 


Index * 


8.9 


2.2 






( 6.7 - 11.8) 


( 1.2 - 


3.9) 


Professional Di plana / Ed. Specialist 




Percent Yes 


4.4 


4.7 




GDR 


7.0 


5.2 






( 5.6 - 8.7) 


( 4.1 - 


6.8) 


Index 


69.8 


62.7 




(56.0 - 87.1) 


(48.2 - 


81.6) 


Associate Degree 
Percent Yes 1 


13.7 


6.7 




GDR 


8.1 


6.9 






( 6.6 - 9.9) 


( 5.5 - 


8.6) 


Index 


36.9 


54.2 




(30.1 - 45.3) 


(43.0 - 


68.2) 


’ Responded "Yes" in original 


interview. 




* Statistically 


significant 


difference between 


1988 and 1991 









In the Teacher survey in both SASS cycles we 
also reinterviewed questions on teaching assignment, 
years in teaching, and plans to remain in teaching (an 
attitude type question). None of these questions 
exhibited significantly improved response variance. 

The teaching assignment questions reinterviewed 
in 1988 and 1991 were similar but not strictly compa- 
rable. In 1991 we reinterviewed a screener question 
used to identify teachers, which asked about full and 
part-time status and included categories for itinerant 
teachers, long-term substitutes, other professional 
staff, and administrators (the last two are out of 
scope for the Teacher survey). The 1988 question 
simply asked about full-time and four levels of part- 
time teaching. The 1988 question includes all full- 
time teachers, the 1991 figure includes only regular 



full-time teachers. These design differences make it 
difficult to compare the two questions, but response 
variance on the number of full-time teachers showed 
no significant change between 1988 and 1991. The 
new categories seem to cause respondents some 
uncertainty — about six percent (s.e. = 0.8) of the 
respondents described their assignment as itinerant 
teacher, long-term substitute, other professional staff, 
or administrator in the original interview. Only three 
percent (s.e. = 0.6) selected one of these answers in 
the reinterview. The data suggest the "itinerant 
teacher" category is the main source of this inconsis- 
tency. It may help to define "itinerant" more clearly 
on future questionnaires. 

The 1988 "years teaching" questions asked, "... 
how many years have you worked as a full-time 
teacher in public and/or private schools ..." (repeated 
for part-time) and provided a cross-tabulation for the 
respondent to complete: 



Pub l ic 
Private 



Years full-time 



Years part-time 



In 1991 we changed the format to ask four 
separate questions: 

• "... how many years have you worked as a full- 
time teacher in private 

• "... part-time in private ...," 

• "... full-time in public ...," and 

• "... part-time in public ..." 



Table 4. Teacher Survey Reinterview Results 
-- Years Teaching 



1988 

Full-tiae, Ptfclic 

GDR 7.6 

( 6.1 - 9.5) 

l- fold Index 10.8 

C 8.7 - 13.4) 

Part-tiae, Ptfclic 

GDR 9.0 

( 6.7 - 12.0) 
l-fold Index 44.4 

(33.2 - 59.3) 

Full-tiae, Private 
GDR 5.2 

( 3.6 - 7.4) 

l-fold Index 12.4 

( 8.7 - 17.7) 

Port-ti«e, Private 
GDR * 3.4 

( 2.1 - 5.8) 

Index 38.5 

(23.0 - 64.4) 



1991 

7.0 

( 5.5 - 8.9) 
C 7 . 7 - 12. A) 



6.6 

( 5.0 • 8.6) 

A2.5 

(32.5 - 55.7) 



5.3 

( 3.3 • 8.7) 

8.8 

( 5. A - 1A.A) 



7.5 

( A. 8 - 11.6) 
37.8 

(2A.A • 58. A) 



* Statistically significant difference between 
1988 and 1991. 
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